Environment recognition system, environment recognition method and robot

ABSTRACT

A system capable of recognizing position, a shape, a posture and the like of an object present in a marginal environment of a device such as a robot in order to make the device perform operations on the object as a subject. In an environment recognition system, 3D information and physical information (color information and the like) of a subject are associated by using camera parameters of each of a 3D image sensor and a 2D image sensor. Thereby, the position, the posture and the shape related to the subject and the physical information of the subject present in the environment of a robot are obtained.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a system and the like recognizing anenvironment of a device.

2. Description of the Related Art

There has been disclosed an approach (for example, refer to JapanesePatent Laid-open No. 2007-322138 referred to as Patent Document 1) forestimating a position of a robot according to image data and range dataacquired respectively from a camera and a laser range finder mounted inthe robot.

However, when the robot performs operations on various objects presentin a marginal environment thereof as a subject according to differentsituations, such as holding an arbitrary object with a hand thereof, itis necessary for it to recognize correctly a position, a shape, aposture and the like of the object.

SUMMARY OF THE INVENTION

The present invention has been accomplished in view of theaforementioned problems, and it is therefore an object of the presentinvention to provide a system and the like capable of recognizing with ahigh degree of accuracy a position, a shape, a posture and the like ofan object present in a marginal environment of a device such as a robotin order to make the device perform operations on the object as asubject.

To attain an object described above, an environment recognition systemaccording to the present invention which is configured to recognize anenvironment of a device comprises: a 3D image sensor configured toacquire 3D information of a subject by photographing a marginalenvironment of the device, a 2D image sensor configured to acquirephysical information of the subject by photographing a range overlappedwith a photographing range of the 3D image sensor, and an imageprocessing element configured to acquire a position, posture and shapeof the subject and the physical information by associating the 3Dinformation of the subject acquired by the 3D image sensor and thephysical information of the subject acquired by the 2D image sensor(First aspect of the present invention).

According to the environment recognition system of the first aspect ofthe present invention, the 3D information of the subject acquiredthrough the 3D image sensor and the physical information acquired by the2D image sensor are associated. Thereby, information needed by a deviceto perform operations on a subject (object) present in the marginalenvironment thereof, namely the position, the posture and the shape ofthe subject and the physical information thereof can be recognized withhigh accuracy. Moreover, since it is preferable to dispose the opticalaxes close to each other in order to make greater the overlapped portionbetween the photographing range of the 3D image sensor and thephotographing range of the 2D image sensor, it is necessary to disposethe two sensors close to each other, thereby, it is expected to make theenvironment recognition system compact.

In the environment recognition system of the first aspect, it isacceptable that the image processing element is configured to calculatea plurality of converted positions as results of converting a pluralityof first pixel positions from a first image coordinate system defined bya photographing area of the 3D image sensor to a second image coordinatesystem defined by a photographing area of the 2D image sensor,respectively, and calculate the 3D information of the subject associatedto the physical information of the subject possessed by second pixelspositioned close to at least three converted positions in the secondimage coordinate system on the basis of the 3D information of thesubject which is possessed by the first pixels and is associatedrespectively to the at least three converted positions in the secondimage coordinate system (Second aspect of the present invention).

In the environment recognition system of the second aspect, it isacceptable that the image processing element is configured to calculatea plane passing through three positions in a global coordinate system asthe 3D information of the subject which is possessed by the first pixelsand is associated respectively to the three converted positions in thesecond image coordinate system, calculate a straight line passingthrough a principle point of the 2D image sensor and the second pixelspositioned close to the three converted positions in the second imagecoordinate system, and calculate a position of an intersection pointbetween the plane and the straight line in the global coordinate systemas the 3D information of the subject associated to the physicalinformation of the subject possessed by the second pixels (Third aspectof the present invention).

In the environment recognition system of the second aspect, it isacceptable that the image processing element is configured to calculatethe physical information of the subject associated to the 3D informationof the subject which is possessed by the first pixels and is associatedrespectively to the converted positions on the basis of the physicalinformation of the subject possessed by one or a plurality of the secondpixels present in a marginal environment of the converted position inthe second image coordinate system (Fourth aspect of the presentinvention).

In the environment recognition system of the first aspect, it isacceptable that the image processing element is configured to calculatea plurality of converted positions as results of converting a pluralityof first pixel positions from a first image coordinate system defined bya photographing area of the 3D image sensor to a second image coordinatesystem defined by a photographing area of the 2D image sensor,respectively, and calculate the physical information of the subjectassociated to the 3D information of the subject which is possessed bythe first pixels and is associated to the converted positions on thebasis of the physical information of the subject possessed by one or aplurality of the second pixels present in the marginal environment ofthe converted position in the second image coordinate system (Fifthaspect of the present invention).

According to the environment recognition system of any of the second tofifth aspects, the 3D information and the physical information areassociated under consideration that the converted positions obtained asa result of converting the first pixels from the first image coordinatesystem to the second image coordinate system usually are not matchedwith the second pixel positions in the second image coordinate system.Thereby, the information needed by the device to perform operations on asubject can be recognized with high accuracy.

In the environment recognition system of the first aspect, it isacceptable that the 2D image sensor is configured to acquire colorinformation or temperature information as the physical information ofthe subject, and the image processing element is configured to acquirethe position, the posture and the shape of the subject, and either oneor both of the color and the temperature information of the subject byassociating the 3D information of the subject acquired through the 3Dimage sensor and either one or both of the color information and thetemperature information of the subject acquired through the 2D imagesensor (Sixth aspect of the present invention).

According to the environment recognition system of the sixth aspect, theinformation needed by the device to perform operations on a subject,namely in addition to the position, the posture and the shape of thesubject, either one of or both of the color information and thetemperature information of the subject can be recognized with highaccuracy.

In the environment recognition system of the first aspect, it isacceptable that the 3D image sensor is configured to acquire luminanceinformation of the subject to be added to the 3D information of thesubject, and the image processing element is configured to performcalibration for obtaining parameters denoting a relationship among the3D image coordinate system, the 2D image coordinate system and theglobal coordinate system by using luminance information and distanceinformation of a calibration object acquired by the 3D image sensor andphysical information of the calibration object acquired by the 2D imagesensor, and to associate the 3D information of the subject acquired bythe 3D image sensor and the physical information of the subject acquiredby the 2D image sensor by using the obtained parameters (Seventh aspectof the present invention).

According to the environment recognition system of the seventh aspect,the calibration for obtaining parameters denoting a relationship betweenthe first image coordinate system and the second image coordinate systemis performed by using luminance information and the 3D information of acalibration object acquired by the 3D image sensor and physicalinformation of the object acquired by the 2D image sensor. Then, the 3Dinformation of the subject acquired by the 3D image sensor and thephysical information of the subject acquired by the 2D image sensor areassociated by using the parameters.

To attain an object described above, an environment recognition methodaccording to the present invention which is configured to recognize anenvironment of a device comprises steps of: acquiring 3D information ofa subject through photographing a marginal environment of the devicewith a 3D image sensor, acquiring physical information of the subjectthrough photographing a range overlapped with a photographing range ofthe 3D image sensor with a 2D image sensor, and acquiring a position,posture and shape of the subject and the physical information of thesubject by associating the 3D information of the subject and thephysical information of the subject (Eighth aspect of the presentinvention).

According to the environment recognition method of the eighth aspect,the information needed by the device to perform operations on a subjectpresent in the marginal environment thereof, namely the position, theposture and the shape of the subject and the physical informationthereof can be recognized with high accuracy.

To attain an object described above, a robot according to the presentinvention which is provided with the environment recognition systemaccording to any of claims 1 to 7 serves as the device configured toperform operations on the subject by using the position, posture andshape of the subject and the physical information recognized by theenvironment recognition system (Ninth aspect of the present invention).

According to the robot of the ninth aspect, since the position, theposture and the shape of the subject and the physical information of thesubject present in the marginal environment thereof can be recognizedwith high accuracy, the robot can perform desired operations on thesubject correctly. Since the system can be made compact, it is possibleto make the entire robot compact or make the space thereof utilizedefficiently.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram illustrating a structure of a robotmounted with an environment recognition system as an embodiment of thepresent invention.

FIG. 2 is a block diagram illustrating the environment recognitionsystem.

FIG. 3 is a flow chart illustrating a processing order of an environmentrecognition method as a first embodiment of the present invention.

FIG. 4 is an explanatory diagram related to a method of matching a threedimensional coordinate system to a two dimensional coordinate system.

FIG. 5 an explanatory diagram related to a method of matching a threedimensional information to a physical information.

FIG. 6 is a flow chart illustrating a processing order of an environmentrecognition method as a second embodiment of the present invention(Example 1).

FIG. 7 is a flow chart illustrating a processing order of an environmentrecognition method as the second embodiment of the present invention(Example 2).

FIG. 8 is an exemplary diagram of a calibration board.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an environment recognition system according to anembodiment of the present invention will be described in detail withreference to the drawings.

First, configurations of the environment recognition system and a robot(device) mounted with the environment recognition will be described. Itshould be noted that the environment recognition system may be mountedin various devices, such as a robot configured to perform manufacturingoperations, transportation operations or the like on an object, and/or avehicle configured to execute an operation control so as to avoidcollision with an object (such as another vehicle in the front).

The robot R illustrated in FIG. 1 is a humanoid robot moving on legs.Similar to a human being, the robot R has a body B0, a head B1 disposedat an upper portion of the body B0, a pair of left and right arms B2extended from both lateral sides of the upper portion of the body B0, ahand H disposed at an end portion of each of the left and right arms B2,a pair of left and right legs B4 extended downward from a lower portionof the body B0. However, the robot R is not limited to a humanoid robot;it may be any type of robots provided with a mechanism equivalent to thearms B2 for changing positions and postures of the hands H.

The robot R is provided with a controller 2 (arithmetic processingelement) configured to control motions thereof. It is acceptable thatthe controller 2 is a distributed control device composed of a maincontrol unit and one or a plurality of sub control units connectedthrough an internal network disposed in the robot R.

The controller 2 is composed of a computer (provided with a CPU,memories such as a ROM, a RAM and the like, and circuits such as an A/Dcircuit, an I/O circuit and the like). A control program is retrievedwhen appropriate from the memories by CPU in the controller, and themotions of the hands H are controlled according to the retrieved controlprogram.

A plurality of actuators 4 are mounted in the robot R. The controller 2controls operations of each of the actuators 4, and consequently,controls the motions of finger mechanisms F1 through F5 of the hand Hand the motions or the like of each joint mechanism in the arm B2 andthe leg B4.

The body B0 is composed of an upper portion and a lower portion joinedin such a way that the two can rotate relatively around the yaw axis.The head B1 can move, for example, rotate around the yaw axis withrespect to the main body B0.

The arm B2 is provided with a first arm link B22 and a second arm linkB24. The body B0 and the first arm link B22 are joined through ashoulder joint mechanism (first arm joint mechanism) B21. The first armlink B22 and the second arm link B24 are joined through an elbow jointmechanism (second arm joint mechanism) B23. The second arm link B24 andthe hand H are joined through a wrist joint mechanism (third arm jointmechanism) B25. The shoulder joint mechanism B21 has degrees of rotationfreedom around the roll axis, the pitch axis and the yaw axis. The elbowjoint mechanism B23 has a degree of rotation freedom around the pitchaxis. The wrist around the pitch axis. The wrist joint mechanism B25 hasdegrees of rotation freedom around the roll axis, the pitch axis and theyaw axis.

The leg B4 is provided with a first leg link B42, a second leg link B44and a foot B5. The main body B0 and the first leg link B42 are joinedthrough a hip joint mechanism (first leg joint mechanism) B41. The firstleg link B42 and the second leg link B44 are joined through a knee jointmechanism (second leg joint mechanism) B43. The second leg link B44 andthe foot B5 are joined through an ankle joint mechanism (third leg jointmechanism) B45.

The hip joint mechanism B41 has degrees of rotation freedom around theroll axis, the pitch axis and the roll axis. The knee joint mechanismB43 has a degree of rotation freedom around the pitch axis. The anklejoint mechanism B45 has degrees of rotation freedom around the roll axisand the pitch axis. The hip joint mechanism B41, the knee jointmechanism B43 and the ankle joint mechanism B45 constitute a “leg jointmechanism group”. The translation and the degree of rotation freedom foreach joint mechanism included in the leg joint mechanism group may bechanged where appropriate. It is acceptable to omit any one jointmechanism in the hip joint mechanism B41, the knee joint mechanism B43and the ankle joint mechanism B45 and constitute the leg joint mechanismgroup with a combination of the remained two joint mechanisms. Moreover,when the leg B4 is provided with a second leg joint mechanism differentfrom the knee joint, the leg joint mechanism group may be constituted byincluding the second leg joint mechanism. In order to relieve impactwhen stepping on floor, the bottom of the foot B5 is disposed with anelastic element B52 as disclosed in Japan Patent Laid-Open No.2001-129774.

The head B1 is installed with a 3D image sensor C1 and a 2D image sensorC2.

In the present embodiment, the 3D image sensor C1 is a TOF (Time OfFlight) 3D sensor. The 3D image sensor C1 acquires 3D information of anobject by photographing the front (a range in the right front of thehead B1) of the robot R.

In the present embodiment, the 2D image sensor C2 is a color sensor. The2D image sensor C2 acquires color information (physical information) ofthe object by photographing a range overlapped with the photographingrange by the 3D image sensor C1.

A resolution of the 3D image sensor (for example, 176×244) is configuredto be different from a resolution of the 2D image sensor (for example,1024×768).

A first image coordinate system is defined according to thephotographing area of the 3D image sensor C1, and a second imagecoordinate system is defined according to the photographing area of the2D image sensor C2. The positions and the postures of the first imagecoordinate system and the second image coordinate system with respect tothe robot coordinate system are stored in memory. As to be describedhereinafter, the position and the posture of the object recognized inthe global coordinate system are used to convert from the globalcoordinate system into the robot coordinate system. In the robotcoordinate system, the mass center of the robot R (for example containedin the body B0), for example, is defined as the origin, the upwarddirection of the robot R is defined as +x direction, the rightwarddirection thereof is defined as +y direction, and the frontwarddirection thereof is defined as +z direction.

As to be described hereinafter, the controller 2 is provided with animage processing element 20 configured to process an image (refer toFIG. 2). The image processing element 20 may be composed of one or aplurality of modules separately.

Functions of the environment recognition system with the above-mentionedconfiguration will be described. Arithmetic processing results arestored in memory and retrieved out of memory where appropriate.

Camera parameters and distortion parameters of the 3D image sensor C1are retrieved from memory (FIG. 3/STEP 102). Camera parameters anddistortion parameters of the 2D image sensor C2 are retrieved frommemory (FIG. 3/STEP 104).

The 3D information of an object is acquired through photographing amarginal environment of the robot R with the 3D image sensor C1 (FIG.3/STEP 106). The 3D information refers to a distance D to the objectfrom the principle point of the sensor C1 at the position of each of aplurality of first pixels in the first image coordinate system.

The color information of an object is acquired through photographing themarginal environment of the robot R with the 2D image sensor C2 (FIG.3/STEP 108). The color information refers to colors (specified by RGBvalues, for example) of the object at the position of each of aplurality of second pixels in the second image coordinate system.

Thereafter, the first pixel position in the first image coordinatesystem is converted to a point in the 3D global coordinate system (FIG.3/STEP 110) by using the camera parameters and the distortion parameterof the 3D image sensors C1.

Specifically, the first pixel position P1 in the first image coordinatesystem is converted to the point Pw in the global coordinate system withthe usage of a rotation matrix Rt and a translation vector Tt whichserve as the camera parameters of the 3D image sensor C1 according tothe relational expression (1). Thereby, as illustrated in FIG. 4, aquantized point (first pixel position) P1=(X1, Y1) in the 3D imagecoordinate system is converted to a point Pw=(xw, yw, zw) in the globalcoordinate system.Pw=Rt ⁻¹(P1−Tt)  (1)

The first pixel position P1=(X1, Y1) can be calculated according to thedistance D from the principle point of the 3D image sensor C1 to thepoint P1, the focal length f, the lens distortion parameters κ1 and κ2,the scale coefficient s_(x) (usually set at “1”) and the followingrelational expressions (2a) to (2c).(d′ _(x) x/s _(x))(1+κ₁ r ²+κ₂ r ⁴)=f(X1/Z1)  (2a)d _(y) y(1+κ₁ r ²+κ₂ r ⁴)=f(Y1/Z1)  (2b)X1² +Y1² +Z1² =D ²  (2c)

Herein, “r” is denoted according to the relational expression (3) byusing an image origin (Cx, Cy) and the like.r=((d′ _(x) x/s _(x))²+(d′ _(y) y)²)^(1/2) ,x≡X1−Cx,y≡Y1−Cy  (3)

“d” denotes the element distance d_(x) in the x direction correctedaccording to the relational expression (4) by using the number ofelements Ncx in the x direction and the number of samples Nfx in thescanning direction.d′ _(x)=(Ncx/Nfx)d _(x)  (4)

Subsequently, a converted position associated to the point Pw in theglobal coordinate system is calculated in the second image coordinatesystem (FIG. 3/STEP 112).

Specifically, a non-linear equation (7) is obtained by using a rotationmatrix R=(R_(ij)) and a translation vector T=(tx, ty, tz) which denotethe camera parameters of the 2D image sensor C2 and the relationalexpressions (5) and (6).x(1+κ₁ r ²+κ₂ r ⁴)=(s _(x) /d′ _(x))f(R ₁₁ xw+R ₁₂ yw+R ₁₃ zw+tx)/(R ₃₁xw+R ₃₂ yw+R ₃₃ zw+tz)  (5)y(1+κ₁ r ²+κ₂ r ⁴)=(1/d _(y))f(R ₂₁ xw+R ₂₂ yw+R ₂₃ zw+ty)/(R ₃₁ xw+R ₃₂yw+R ₃₃ zw+tz)  (6)r+κ ₁ r ³+κ₂ r ⁵ =c=(c1² +c2²)^(1/2).c1≡f(R ₁₁ xw+R ₁₂ yw+R ₁₃ zw+tx)/(R ₃₁ xw+R ₃₂ yw+R ₃₃ zw+tz),c2≡f(R ₂₁ xw+R ₂₂ yw+R ₂₃ zw+ty)/(R ₃₁ xw+R ₃₂ yw+R ₃₃ zw+tz)  (7)

The relational expression (8) is obtained by approximating thenon-linear equation (7) according to Newton-Raphson method.f(r _(n))=r _(n)+κ₁ r _(n) ³+κ₂ r _(n) ⁵ −c,f′(r _(n))=1+3κ₁ r _(n)²+5κ₂ r _(n) ⁴ ,r _(n)+1=r _(n) f(r _(n))/f′(r _(n))  (8)

r_(n) is calculated iteratively according to the relational expression(8), and the convergence result when the convergence rate|1−(r_(n+1)/r_(n))| thereof is equal to or smaller than the threshold ε(for example, exp(−10)) is calculated as an approximate solution of r.

The converted position Pc=(xc, yc) in the second image coordinate systemis calculated according to the relational expressions (3), (5) and (6)by using the approximate solution r. Thereby, as illustrated in FIG. 4,the converted position Pc in the second image coordinate systemcorresponded to the point Pw=(xw, yw, zw) in the global coordinatesystem can be obtained.

As aforementioned, the resolution of the 3D image sensor C1 and theresolution of the 2D image sensor C2 are different from each other;however, it is necessary to perform interpolation despite whether or notthere is a difference between the resolutions. The reason is that asillustrated in FIG. 4, normally, the converted position Pc in the secondimage coordinate system corresponded to the first pixel position (X1,Y1) in the first image coordinate system does not match the quantizedpoint (the second pixel position) in the second image coordinate system.In FIG. 4, the converted position Pc in the second image coordinatesystem is contained in a rectangular area with the second pixelpositions of A to D as the apexes.

In this situation, the color information (physical information) isassociated to the 3D information according to, for example, the nearestneighbor interpolation, the bilinear interpolation or the bi-cubicinterpolation method (FIG. 3/STEP 114).

According to the nearest neighbor interpolation method, the colorinformation of the second pixel position A which is closest to theconverted position Pc in the second image coordinate system is appliedto the converted position Pc as the color information. According to thebilinear interpolation method, the color information of the convertedposition Pc is determined on the basis of a ratio of each distance fromthe converted position Pc to each of the second pixel positions of A toD enclosing the converted position Pc in the second image coordinatesystem and the color information of each of the second pixel positionsof A to D. According to the bi-cubic interpolation method, the colorinformation of the converted position Pc is determined on the basis of atotal sum of 16 second pixel position including the 4 second pixelpositions of A to D enclosing the converted position Pc in the secondimage coordinate system and further the 12 second pixel positions outerof the 4 second pixel positions of A to D, and the color information ofeach of the 16 second pixel positions.

Thereafter, whether or not the coordinate conversion to the globalcoordinate system, the determination of the corresponding convertedposition in the second image coordinate system, the calculation of thecolor information according to the nearest neighbor interpolation methodor the like, and the association of the physical information (colorinformation) to the 3D information mentioned above have been completedfor a first pixel position (not necessary to be performed on all thefirst pixel positions) serving as a subject in the first imagecoordinate system is determined (FIG. 3/STEP 116).

If the determination result is negative (FIG. 3/STEP 116 . . . NO), thecoordinate conversion to the global coordinate system, the determinationof the corresponding converted position in the second image coordinatesystem, the calculation of the physical information (color information)according to the nearest neighbor interpolation method or the like, andthe association of the physical information to the 3D information areiteratively performed on the remained first pixel positions serving asthe subject (FIG. 3/STEP 110 to STEP 114).

On the other hand, if it is determined that the mentioned process hasbeen performed on all the first pixel positions serving as the subject(FIG. 3/STEP 116 . . . YES), whether or not it is necessary to associatethe 3D information to the 2D image is determined according to, forexample, whether the resolution of the 3D image sensor C1 is greaterthan the resolution of the 2D image sensor C2 (FIG. 3/STEP 118). It isacceptable to omit the determination processing.

If the determination result is affirmative (FIG. 3/STEP 118 . . . YES),as illustrated in FIG. 5, the coordinate values of Pw1=(xw1, yw1, zw1),Pw2=(xw2, yw2, zw2) and Pw3=(xw3, yw3, zw3) in the global coordinatesystem corresponded respectively to the 3 converted positions Pc1, Pc2and Pc3 which are the closet points to enclose the second pixel positionP2=(X2, Y2) serving as the subject in the second image coordinate systemare retrieved from memory (FIG. 3/STEP 120). Then, whether or not the 3points of Pw1 Pw2 and Pw3 in the global coordinate system are in thesame straight line is confirmed.

Thereafter, as illustrated in FIG. 5, a plane passing through the 3points of Pw1, Pw2 and Pw3 in the global coordinate system is determined(STEP 3/STEP 122).

The plane passing through the 3 points of Pw1, Pw2 and Pw3 in the globalcoordinate system is donated by the relational expression (9).u1xw+u2yw+u3zw=1,^(t)(u1,u2,u3)=Q ⁻¹·^(t)(1,1,1),Q≡ ^(t)(^(t) Pw1,^(t) Pw2,^(t) Pw3)  (9)

Subsequently, a straight line passing through the principle point of the2D image sensor C2 and the converted position Pc in the second imagecoordinate system is determined, and thereafter, an intersection pointbetween the plane and the straight line is determined as the point Pw inthe global coordinate system corresponded to the converted position Pc(FIG. 3/STEP 124).

The converted position pc=(Xc, Ye) in the second image coordinate systemis denoted by the relational expressions (10) and (11).Xc(1+κ₁ r ²+κ₂ r ⁴)=(s _(x) /d′ _(x))f(r ₁₁ Xw+r ₁₂ Xw+r ₁₃ Xw+tx)/(r ₃₁Xw+r ₃₂ Yw+r ₃₃ Zw+tz) (r=(Xc)²+(Yc)²)^(1/2))  (10)Yc(1+κ₁ r ²+κ₂ r ⁴)=(1/d _(y))f(r ₂₁ Xw+r ₂₂ Yw+r ₂₃ Zw+ty)/(r ₃₁ Xw+r₃₂ Yw+r ₃₃ Zw+tz)  (11)

The coordinate value of Pw=(Xw, Yw, Zw) in the global coordinate systemcorresponded to the converted position is calculated according to therelational express (12) obtained from the relational expressions of (9)to (11).¹(Xw,Yw,Zw)=B ⁻¹·^(t)(1,1,1),B≡(Bij),B _(1j) ={Xc(1+κ₁ r ²+κ₂ r ⁴)r _(3j)−(s _(x) /d′ _(x))f×r _(ij)}/{(s_(x) /d′ _(x))f×tx−Xc(1+κ₁ r ²+κ₂ r ⁴)×tz}. B _(3j) =uj  (12)

Thereafter, whether or not the selection of the 3 converted positions in2D coordinate system to the global coordinate system, the determinationof the plane in the global coordinate system, the determination of thestraight line passing through the principle point of the 2D image sensorC2 and a second pixel position P2 (not necessary to be performed on allthe second pixel positions) and the determination of the intersectionpoint between the plane and the straight line have been completed forthe second pixel position P2 serving as a subject in the second imagecoordinate system is determined (FIG. 3/STEP 126).

If the determination result is negative (FIG. 3/STEP 126 . . . NO), theselection of the 3 converted positions in 2D coordinate system to theglobal coordinate system, the determination of the plane in the globalcoordinate system, the determination of the straight line passingthrough the principle point of the 2D image sensor C2 and the secondpixel position and the determination of the intersection point betweenthe plane and the straight line are iteratively performed on theremained second pixel positions serving as the subject (FIG. 3/STEP 120to STEP 124).

On the other hand, if it is determined that the mentioned process hasbeen performed on all the second pixel positions serving as the subject(FIG. 3/STEP 126 . . . YES), whether or not it is necessary that theacquisition of the 3D information by the 3D image sensor C1 (refer toFIG. 3/STEP 106) and the acquisition of the physical information throughthe 21) image sensor C2 (refer to FIG. 3/STEP 108), and the subsequentiterative processing performed thereon is determined (FIG. 3/STEP 128).If it is determined that it is not necessary to associate the 3Dinformation to the 2D image (FIG. 3/STEP 118 . . . NO), whether theiterative processing is necessary or not is determined (FIG. 3/STEP128).

If it is determined that the iterative processing is necessary (FIG.3/STEP 128 . . . NO), the acquisition of the 3D information by the 3Dimage sensor C1, the acquisition of the physical information through the2D image sensor C2 and the subsequent processing performed thereon arecarried out iteratively (refer to FIG. 3/STEP 106 to STEP 126).

If it is determined that the iterative processing is unnecessary (FIG.3/STEP 128 . . . YES), the aforementioned series of processing areterminated.

According thereto, the position, posture and shape of an object in theglobal coordinate system and colors of each point in the globalcoordinate system, namely information related to the object which isnecessary for the robot R to hold the object, are obtained. It is alsoacceptable to retrieve additional information of the object, such as thetype, the mass center or the like of the object, which can be obtainedthrough database searching, pattern matching or the like on the basis ofthe shape or the like of the object if necessary.

According to the environment recognition system exhibiting theabove-mentioned functions, information related to an object present in amarginal environment of the robot R, namely a position, a posture and ashape of the object in the global coordinate system, and color of eachpoint in the global coordinate system can be recognized with highaccuracy. Thereby, the robot R can perform operations on the objectserving as a subject with certain, such as holding the object or thelike by moving the arm B2, the hand H and finger mechanisms of F1 to F5,respectively. Additionally, the robot R may adjust the position(position of the mass center of the main body B0) and the posture (angleof each axis of the main body coordinate system with respect to eachaxis of the global coordinate system) of the main body B0 by moving theleg B4 or the like before performing operations if necessary.

In order to enlarge the overlapped area between the photographing rangeof the 3D image sensor C1 and the photographing range of the 2D imagesensor C2, it is desired to make the optical axes thereof closer, andconsequently, the sensors C1 and C2 are disposed closer to make thesystem compact. If the system is made compact, it is possible to makethe entire robot R compact or make the space thereof utilizedefficiently.

As the 3D image sensor C1 for acquiring the 3D information, in additionto the TOF 3D image sensor, it is acceptable to use a part of or theentire part of a TOF 3D image sensor of a scanning type and a stereocamera. As the 2D image sensor C2 for acquiring the physicalinformation, in addition to the visual light color camera to acquire thecolor of a subject, it is acceptable to use a part of or the entire partof a visual light monochrome camera to acquire shading information ofthe subject, an infrared camera to acquire night vision information ofthe subject, a far infrared camera to acquire temperature information ofthe subject, a millimeter wave camera to acquire millimeter wave sourceinformation, and a polarization camera to acquire information in thenormal line of a plane.

Various combinations of the mentioned 3D image sensor C1 and the 2Dimage sensor C2 can be adopted. For example, as the constituent elementof the environment system, the combination may be (1) one 3D imagesensor C1 and one 2D image sensor C2, (2) one 3D image sensor C1 and aplurality of 2D image sensors C2 of the same type or different types,(3) a plurality of 3D image sensors C3 of the same type or differenttypes and one 2D image sensors C2, or (4) a plurality of 3D imagesensors C3 of the same type or different types and a plurality of 2Dimage sensors C2 of the same type or different types.

As another embodiment, descriptions will be carried out on imageprocessing procedure when two 3D image sensors C1 (TOF 3D image sensor)and two 2D image sensors C2 of different types (one is visual lightcolor camera and the other is far infrared camera) is adopted in theenvironment recognition system.

Firstly, the color parameters and the distortion parameters of each ofthe two 3D image sensors C1 are retrieved from memory (FIG. 6/STEP 202).

The color parameters and the distortion parameters of each of the two 2Dimage sensors C2 are retrieved from memory (FIG. 6/STEP 204).

The 3D information of a subject is obtained through photographing themarginal environment of the robot R with the two 3D image sensors C1,respectively (FIG. 6/STEP 206).

The physical information, namely the color information and thetemperature information of the subject are obtained throughphotographing the marginal environment of the robot R with the colorcamera and the far infrared camera serving as the 2D image sensors C2,respectively (FIG. 6/STEP 208 and STEP 210).

Thereafter, the first pixel position in the first image coordinatesystem (2D coordinate system) of the first 3D image sensor C11 isconverted to a point in the global coordinate system (3D coordinatesystem) by using the camera parameters and the distortion parameter ofthe first 3D image sensor C11 (FIG. 6/STEP 212).

Subsequently, a converted position Pc11 in the second image coordinatesystem of the color camera, which corresponds to the point Pw1=(xw1,yw1, zw1) in the global coordinate system, is calculated (FIG. 6/STEP214).

Then, according to the nearest neighbor interpolation method or thelike, the color information (R, G, B) of the converted position Pc11 iscalculated, and the color information (R, G, B) is associated to the 3Dinformation of the first pixel position P111 in the first imagecoordinate system of the first 3D image sensor C11 which corresponds tothe converted position Pc 11 (FIG. 6/STEP 216 (refer to FIG. 4)).

Further, a converted position Pc 12 in the second image coordinatesystem of the far infrared camera, which corresponds to the point Pw1 inthe global coordinate system, is calculated (FIG. 6/STEP 218).

Then, according to the nearest neighbor interpolation method or thelike, the temperature information Temp (or luminance information Lumrelated to the temperature) of the converted position Pc12 iscalculated, and the temperature information Temp is associated to the 3Dinformation of the first pixel position P112 in the first imagecoordinate system of the first 3D image sensor C11 which corresponds tothe converted position Pc12 (FIG. 6/STEP 220 (refer to FIG. 4)).

Thereafter, whether or not the coordinate conversion to the globalcoordinate system, the determination of the corresponding convertedposition in the second image coordinate system, the calculation of thephysical information (the color information and the temperatureinformation) according to the nearest neighbor interpolation method orthe like, and the association of the physical information to the 3Dinformation as mentioned above have been completed for a first pixelposition (not necessary to be performed on all the first pixelpositions) serving as a subject in the first image coordinate system ofthe first 3D image sensor C11 is determined (FIG. 6/STEP 222).

If the determination result is negative (FIG. 6/STEP 222 . . . NO), thecoordinate conversion to the global coordinate system, the determinationof the corresponding converted position in the second image coordinatesystem, the calculation of the physical information (the colorinformation and the temperature information) according to the nearestneighbor interpolation method or the like, and the association of thephysical information to the 3D information are iteratively performed onthe remained first pixel positions serving as the subject (FIG. 6/STEP212 to STEP 220).

On the other hand, if it is determined that the mentioned process hasbeen performed on all the first pixel positions serving as the subjectin the first image coordinate system of the first 3D image sensor C11(FIG. 6/STEP 222 . . . YES), the first pixel position in the first imagecoordinate system (2D coordinate system) of the second 3D image sensor C12 is coordinate converted to a point in the global coordinate system(3D coordinate system) by using the camera parameters and the distortionparameter of the second 3D image sensor C12 (FIG. 6/STEP 224).

Subsequently, a converted position Pc21 in the second image coordinatesystem of the color camera, which corresponds to the point Pw2=(xw2,yw2, zw2) in the global coordinate system, is calculated (FIG. 6/STEP226).

Then, according to the nearest neighbor interpolation method or thelike, the color information (R, G, B) of the converted position Pc21 iscalculated, and the color information (R, G, B) is associated to the 3Dinformation of the first pixel position P121 in the first imagecoordinate system of the second 3D image sensor C12 which corresponds tothe converted position Pc21 (FIG. 6/STEP 228 (refer to FIG. 4)).

Further, a converted position Pc22 in the second image coordinate systemof the far infrared camera, which corresponds to the point Pw2 in theglobal coordinate system, is calculated (FIG. 6/STEP 230).

Then according to the nearest neighbor interpolation method or the like,the temperature information Temp (or luminance information Lum relatedto the temperature) of the converted position Pc22 is calculated, andthe temperature information Temp is associated to the 3D information ofthe first pixel position P122 in the first image coordinate system ofthe second 3D image sensor C12 which corresponds to the convertedposition Pc22 (FIG. 6/STEP 232 (refer to FIG. 4)).

Thereafter, whether or not the coordinate conversion to the globalcoordinate system, the determination of the corresponding convertedposition in the second image coordinate system, the calculation of thephysical information (the color information and the temperatureinformation) according to the nearest neighbor interpolation method orthe like, and the association of the physical information to the 3Dinformation as mentioned above have been completed for a first pixelposition (not necessary to be performed on all the first pixelpositions) serving as a subject in the first image coordinate system ofthe second 3D image sensor C12 is determined (FIG. 6/STEP 234).

If the determination result is negative (FIG. 6/STEP 234 . . . NO), thecoordinate conversion to the global coordinate system, the determinationof the corresponding converted position in the second image coordinatesystem, the calculation of the physical information (the colorinformation and the temperature information) according to the nearestneighbor interpolation method or the like, and the association of thephysical information to the 3D information are iteratively performed onthe remained first pixel positions serving as the subject (FIG. 6/STEP224 to STEP 232).

On the other hand, if it is determined that the mentioned process hasbeen performed on all the first pixel positions serving as the subjectin the first image coordinate system of the second 3D image sensor C12(FIG. 6/STEP 234 . . . YES), whether or not it is necessary to associatethe 3D information to the 2D image is determined (FIG. 7/STEP 236).

If the determination result is affirmative (FIG. 7/STEP 236 . . . YES),the coordinate values of three points in the global coordinate systemcorresponded respectively to the 3 converted positions which are thecloset points to enclose the second pixel position P21 in the secondimage coordinate system of the color camera are retrieved from memory(FIG. 7/STEP 240). Thereafter, a plane passing through the 3 points inthe global coordinate system is determined (STEP 7/STEP 242 (refer toFIG. 5 and the relational expression (9))).

Subsequently, an optical axis passing through the principle point of thecolor camera and the second pixel position P21 in the second imagecoordinate system of the color camera is determined, and thereafter, anintersection point between the plane and the optical axis is determinedas the point Pw in the global coordinate system corresponded to theconverted position Pc (FIG. 7/STEP 244 (refer to FIG. 5 and therelational expression (12))).

Thereafter, whether or not the selection of the 3 converted positions in2D coordinate system to the global coordinate system, the determinationof the plane in the global coordinate system, the determination of theoptical axis passing through the principle point of the color camera anda second pixel position (not necessary to be performed on all the secondpixel positions) and the determination of the intersection point betweenthe plane and the optical axis have been completed for the second pixelposition serving as a subject in the second image coordinate system ofthe color camera is determined (FIG. 7/STEP 246).

If the determination result is negative (FIG. 7/STEP 246 . . . NO), theselection of the 3 converted positions in 2D coordinate system to theglobal coordinate system, the determination of the plane in the globalcoordinate system, the determination of the optical axis passing throughthe principle point of the color camera and the second pixel position,and the determination of the intersection point between the plane andthe optical axis are iteratively performed on the remained second pixelpositions serving as the subject in the second image coordinate systemof the color camera (FIG. 7/STEP 240 to STEP 244).

On the other hand, if it is determined that the mentioned process hasbeen performed on all the second pixel positions serving as the subjectin the second image coordinate system of the color camera (FIG. 7/STEP246 . . . YES), the coordinate values of points in the global coordinatesystem corresponded respectively to the converted positions which arethe closet points to enclose the second pixel position P22 serving asthe subject in the second image coordinate system of the far infraredcamera are retrieved from memory (FIG. 7/STEP 248). Thereafter, a planepassing through the 3 points in the global coordinate system isdetermined (STEP 7/STEP 250 (refer to FIG. 5 and the relationalexpression (9))).

Subsequently, a straight line passing through the principle point of thefar infrared camera and the second pixel position P22 in the secondimage coordinate system of the far infrared camera is determined, andthereafter, an intersection point between the plane and the straightline is determined as the point Pw in the global coordinate systemcorresponded to the converted position Pc (FIG. 7/STEP 252 (refer toFIG. 5 and the relational expression (12))).

Thereafter, whether or not the selection of the 3 converted positions in2D coordinate system to the global coordinate system, the determinationof the plane in the global coordinate system, the determination of thestraight line passing through the principle point of the far infraredcamera and a second pixel position (not necessary to be performed on allthe second pixel positions) and the determination of the intersectionpoint between the plane and the straight line have been completed forthe second pixel position serving as a subject in the second imagecoordinate system of the far infrared camera is determined (FIG. 7/STEP254).

If the determination result is negative (FIG. 7/STEP 254 . . . NO), theselection of the 3 converted positions in 2D coordinate system to theglobal coordinate system, the determination of the plane in the globalcoordinate system, the determination of the straight line passingthrough the principle point of the far infrared camera and the secondpixel positions and the determination of the intersection point betweenthe plane and the straight line are iteratively performed (FIG. 7/STEP248 to STEP 252).

On the other hand, if it is determined that the mentioned process hasbeen performed on all the second pixel positions serving as the subjectin the second image coordinate system of the far infrared camera (FIG.7/STEP 254 . . . YES), whether or not it is necessary that theacquisition of the 3D information by each of the two 3D image sensors C1(refer to FIG. 6/STEP 206) and the acquisition of the physicalinformation through each of the two 2D image sensors C2 (refer to FIG.6/STEP 208 and STEP 210), and the subsequent iterative processingperformed thereon is determined (FIG. 7/STEP 256). If it is determinedthat it is not necessary to associate the 3D information to the 2D image(FIG. 7/STEP 236 . . . NO), whether the iterative processing isnecessary or not is determined (FIG. 7/STEP 256).

If it is determined that the iterative processing is necessary (FIG.7/STEP 256 . . . NO), the acquisition of the 3D information by each ofthe two 3D image sensors C1, the acquisition of the physical informationby each of the two 2D image sensors C2 and the subsequent processingperformed thereon are carried out iteratively (refer to FIG. 6/STEP 206to STEP 234, FIG. 7/STEP 236 to STEP 254).

If it is determined that the iterative processing is unnecessary (FIG.7/STEP 256 . . . YES), the aforementioned series of image processing areterminated.

According thereto, the information related to the object which isnecessary for the robot R to hold the object, namely, the position,posture and shape of an object in the global coordinate system and thephysical information of each point of the object (color and temperature)in the global coordinate system, are obtained.

Note that it is acceptable to perform calibration according to anapproach to be described hereinafter to acquire the camera parameters.

Firstly, a 3D image composed of pixels having luminance information anddistance information is obtained by photographing a calibration board(an object used for calibration) in the marginal environment of therobot R through the usage of the 3D image sensor C1.

By photographing the calibration board through the usage of the 2D imagesensor C2 in a range overlapped with the photographing range of the 3Dimage sensor C1, a 2D image composed of pixels having color informationis obtained.

Camera parameters are determined according to the calibration. Thecalibration is performed according to common approaches (refer to R. Y.Tsai: “An Efficient and accurate calibration technique for 3D machinevision” (CVPR, pp. 364-374 (1986)), and R. Y. Tsai: “A versatile cameracalibration technique for high-accuracy 3D machine vision metrologyusing off-the-shelf TV cameras and lenses” (J-RA, vol. 3, pp. 323-344(1987))).

Herein, as illustrated in FIG. 8, the calibration board used has squaresin black and white colors arranged alternatively to a checkered pattern,and there is a triangular mark in the corner of a white square positionclose to the center of the board. The mark is used to confirm whether ornot the apexes of the plurality of squares of the board taken by onecamera correspond to the apexes of the plurality of squares of the boardtaken by the other camera.

The formation of the pattern may be varied according to the type of theimage sensor serving as the calibration subject. For example, if the 2Dimage sensor C2 is an active near infrared camera, the aforementionedpattern may be formed according to the presence of coatings whichreflect near infrared rays.

The mark is formed by coating infrared-reflecting coatings or fixing aninfrared-reflecting seal on the board. Thereby, it can be expected toimprove the recognition accuracy of the mark via the 3D image sensor.The surface of the board is processed with white squares to preventdiffused reflection and with black squares to induce diffusedreflection. Thereby, it can be expected to improve the recognitionaccuracy of the black-white pattern via the 3D image sensor.

External parameters and internal parameters of the camera are calculatedaccording to global coordinates obtained through calibration andcorresponded image coordinates of the camera and the relationalexpressions (3), (5) and (6). The rotation matrix R (tertiary squarematrix having three unknown quantities) and the translation vector T(having three unknown quantities) are calculated as the externalparameters. The focal length f, the lens distortion parameters κ₁ andκ₂, the scale coefficient s_(x) (which is normally set equal at “1”) andthe origin of the image (Cx, Cy) are determined as the internalparameters. The camera parameters obtained through calibration are storein memory.

1. An environment recognition system configured to recognize anenvironment of a device, comprising: a 3D image sensor configured tophotograph a marginal environment of the device and to acquire 3Dinformation of a subject by photographing the marginal environment ofthe device, a 2D image sensor configured to photograph a rangeoverlapped with a photographing range of the 3D image sensor and toacquire physical information of the subject by photographing the rangeoverlapped with the photographing range of the 3D image sensor, and animage processing element configured to acquire a position, posture andshape of the subject and the physical information by associating the 3Dinformation of the subject acquired by the 3D image sensor and thephysical information of the subject acquired by the 2D image sensor,wherein the 3D image sensor is independent from the 2D image sensor, anda resolution of the 3D image sensor is different from a resolution ofthe 2D image sensor.
 2. The environment recognition system according toclaim 1, wherein the image processing element is configured to calculatea plurality of converted positions as results of coordinate converting aplurality of first pixel positions from a first image coordinate systemdefined by a photographing area of the 3D image sensor to a second imagecoordinate system defined by a photographing area of the 2D imagesensor, respectively, and calculate the 3D information of the subjectassociated to the physical information of the subject possessed bysecond pixels positioned close to at least three converted positions inthe second image coordinate system on the basis of the 3D information ofthe subject which is possessed by the first pixels and is associatedrespectively to the at least three converted positions in the secondimage coordinate system.
 3. The environment recognition system accordingto claim 2, wherein the image processing element is configured tocalculate a plane passing through three positions in a global coordinatesystem as the 3D information of the subject which is possessed by thefirst pixels and is associated respectively to the three convertedpositions in the second image coordinate system, calculate a straightline passing through a principle point of the 2D image sensor and thesecond pixels positioned close to the three converted positions in thesecond image coordinate system, and calculate a position of anintersection point between the plane and the straight line in the globalcoordinate system as the 3D information of the subject associated to thephysical information of the subject possessed by the second pixels. 4.The environment recognition system according to claim 2, wherein theimage processing element is configured to calculate the physicalinformation of the subject associated to the 3D information of thesubject which is possessed by the first pixels and is associatedrespectively to the converted positions on the basis of the physicalinformation of the subject possessed by one or a plurality of the secondpixels present in a marginal environment of the converted position inthe second image coordinate system.
 5. The environment recognitionsystem according to claim 1, wherein the image processing element isconfigured to calculate a plurality of converted positions as results ofconverting a plurality of first pixel positions from a first imagecoordinate system defined by a photographing area of the 3D image sensorto a second image coordinate system defined by a photographing area ofthe 2D image sensor, respectively, and calculate the physicalinformation of the subject associated to the 3D information of thesubject which is possessed by the first pixels and is associated to theconverted positions on the basis of the physical information of thesubject possessed by one or a plurality of the second pixels present inthe marginal environment of the converted position in the second imagecoordinate system.
 6. The environment recognition system according toclaim 1, wherein the 2D image sensor is configured to acquire colorinformation or temperature information as the physical information ofthe subject, and the image processing element is configured to acquirethe position, the posture and the shape of the subject, and either oneor both of the color and the temperature information of the subject byassociating the 3D information of the subject acquired through the 3Dimage sensor and either one or both of the color information and thetemperature information of the subject acquired through the 2D imagesensor.
 7. The environment recognition system according to claim 1,wherein the 3D image sensor is configured to acquire luminanceinformation of the subject to be added to the 3D information of thesubject, and the image processing element is configured to performcalibration for obtaining parameters denoting a relationship among the3D image coordinate system, the 2D image coordinate system and theglobal coordinate system by using luminance information and distanceinformation of a calibration object acquired by the 3D image sensor andphysical information of the calibration object acquired by the 2D imagesensor, and to associate the 3D information of the subject acquired bythe 3D image sensor and the physical information of the subject acquiredby the 2D image sensor by using the obtained parameters.
 8. Anenvironment recognition method which is configured to recognize anenvironment of a device, comprising steps of: photographing a marginalenvironment of the device with a 3D image sensor and acquiring 3Dinformation of a subject through the photographing of the marginalenvironment of the device with a 3D image sensor having a firstresolution, photographing a range overlapped with a photographing rangeof the 3D image sensor with a 2D image sensor, which is independent fromthe 3D image sensor, and acquiring physical information of the subjectthrough the photographing of the range overlapped with the photographingrange of the 3D image sensor with the 2D image sensor having a secondresolution different from the first resolution, and acquiring aposition, posture and shape of the subject and the physical informationof the subject by associating the 3D information of the subject and thephysical information of the subject.
 9. A robot provided with anenvironment recognition system, wherein the environment recognitionsystem is provided with a 3D image sensor configured to photograph amarginal environment of the robot and to acquire 3D information of asubject by photographing the marginal environment of the robot, a 2Dimage sensor configured to photograph a range overlapped with aphotographing range of the 3D image sensor and to acquire physicalinformation of the subject by photographing the range overlapped withthe photographing range of the 3D image sensor, and an image processingelement configured to acquire a position, posture and shape of thesubject and the physical information of the subject by associating the3D information of the subject acquired by the 3D image sensor and thephysical information of the subject acquired by the 2D image sensor,wherein the 3D image sensor is independent from the 2D image sensor, anda resolution of the 3D image sensor is different from a resolution ofthe 2D image sensor, and the robot is configured to perform operationson the subject by using the position, posture and shape of the subjectand the physical information recognized by the environment recognitionsystem.
 10. The environment recognition system according to claim 1,wherein the 3D image sensor is a time of flight 3D sensor.
 11. Theenvironment recognition system according to claim 10, wherein the 2Dimage sensor is at least one of a light color camera and a far infraredcamera.
 12. The environment recognition system according to claim 1,wherein said range overlapped with the photographing range of the 3Dimage sensor is partially different from the photographing range of the3D image sensor.
 13. The environment recognition system according toclaim 6, wherein the 2D image sensor is a far infrared camera, and the2D image sensor is configured to acquire only temperature information asthe physical information of the subject, and the image processingelement is configured to acquire the position, the posture and the shapeof the subject, and the temperature information of the subject byassociating the 3D information of the subject acquired through the 3Dimage sensor and the temperature information of the subject acquiredthrough the 2D image sensor.
 14. The robot according to claim 9, whereinthe 3D image sensor is a time of flight 3D sensor.
 15. The robotaccording to claim 14, wherein the 2D image sensor is at least one of alight color camera and a far infrared camera.
 16. The robot according toclaim 9, wherein said range overlapped with the photographing range ofthe 3D image sensor is partially different from the photographing rangeof the 3D image sensor.
 17. The robot according to claim 9, wherein the2D image sensor is a far infrared camera configured to acquiretemperature information as the physical information of the subject, andthe image processing element is configured to acquire the position, theposture and the shape of the subject, and the temperature information ofthe subject by associating the 3D information of the subject acquiredthrough the 3D image sensor and the temperature information of thesubject acquired through the 2D image sensor.
 18. The method accordingto claim 8, wherein said range overlapped with the photographing rangeof the 3D image sensor is partially different from the photographingrange of the 3D image sensor.